Just as you can add to a selection, you can also subtract using any of the selection tools. As long as you have not moved your selection or changed it in any way, you can subtract from the original selection area by holding down the Ctrl + Shift keys. Simply choose a selection tool, hold down the Ctrl + Shift keys at the same time, and draw into the area of the original selection that you would like to remove.
The small minus sign inside the selection cursor indicates that instead of creating a new selection, you are subtracting from an existing selection.
Subtracting from a rectangle selection
Feel free to mix and match selection tools. Even if the original selection was created using the rectangle selection tool, you can subtract from it with the Oval, Magic Wand, Polygon, or Lasso tools.